fix(index): isolate null-only zone maps in version 1 - #8190

Are you saying that version 1 is only used for the null-only case? In other words, a writer will choose version 0 or version 1 based on the data type?

I think we want version numbers to be more of an increasing, inclusive concept. In other words...

Version 0 does not know how to create null-only zone maps.
Version 1 can create everything version 0 can and can also do null-only zone maps.

Is this a correct understanding?

Yes, that is the intended compatibility model. ZoneMapIndexPlugin::version() reports the maximum version this implementation supports (1), while each created index records the minimum reader version required by its physical layout. Ordered indices therefore continue to write version 0, and null-only indices write version 1. A version-1 implementation can read and create both layouts; a version-0 reader retains ordered indices and ignores null-only indices so it falls back to scanning.

Why do we need to store the data type in the index?

Null-only zone maps store min and max physically as Arrow Null, so their file schema no longer carries the indexed logical type. The scalar-index loader receives the index store and details, but not the dataset field, and the logical type is needed after loading to validate updates and rebuild the correct processor/seeds. The data_type global buffer preserves it as a one-field Arrow IPC schema. Ordered version-0 maps still infer the type from min/max and do not write this buffer.

What is null_offsets here?

null_offsets contains the exact top-level-null row positions relative to the start of each seed zone, encoded as little-endian u64 values in the binary field. null_count alone is insufficient to reconstruct the complete RowAddrTreeMap when an append/update is built from seeds. During seed loading, each zone-relative offset is combined with the zone start and fragment ID to recover the absolute row address. Legacy seeds without this field deliberately fall back to the scanned update path.

This fallback indexes the batch by name before try_from_serialized_with_mode reaches its fallible schema checks. RecordBatch string indexing unwraps a missing column, so an otherwise parseable v0 index without min now panics the process; the same file returned Err at the base revision. Use a fallible column_by_name("min") lookup for ordered fallback, and require/validate data_type for null-only mode instead of silently adopting physical Null.

#[tokio::test]
async fn test_missing_min_returns_error() {
    let tmpdir = TempObjDir::default();
    let store = Arc::new(LanceIndexStore::new(
        Arc::new(ObjectStore::local()),
        tmpdir.clone(),
        Arc::new(LanceCache::no_cache()),
    ));
    let schema = Arc::new(Schema::new(vec![
        Field::new("max", DataType::Int32, true),
        Field::new("null_count", DataType::UInt32, false),
        Field::new("nan_count", DataType::UInt32, false),
        Field::new("fragment_id", DataType::UInt64, false),
        Field::new("zone_start", DataType::UInt64, false),
        Field::new("zone_length", DataType::UInt64, false),
    ]));
    let batch = RecordBatch::try_new(schema.clone(), vec![
        Arc::new(Int32Array::from(vec![Some(99)])) as _,
        Arc::new(UInt32Array::from(vec![0])) as _,
        Arc::new(UInt32Array::from(vec![0])) as _,
        Arc::new(UInt64Array::from(vec![0])) as _,
        Arc::new(UInt64Array::from(vec![0])) as _,
        Arc::new(UInt64Array::from(vec![1])) as _,
    ]).unwrap();
    let mut writer = store.new_index_file(ZONEMAP_FILENAME, schema).await.unwrap();
    writer.write_record_batch(batch).await.unwrap();
    writer.finish().await.unwrap();

    let result = ZoneMapIndex::load(store, None, &LanceCache::no_cache(), false).await;
    assert!(result.is_err(), "a malformed index must return an error");
}